This issue developed on a 2020 5K iMac after I updated to Monterey 12.0.1 public release, then to the 12.1 Beta, then did a full wipe and a clean reinstall of Big Sur 11.6.1. Never had it before (this iMac shipped with Catalina and I updated to Big Sur without doing a clean install).
In my case it starts happening on the second or third day after a restart. Once it starts happening, if I unplug the USB audio interface, the whole computer locks up and needs a hard reset.